<!DOCTYPE html>
<html lang="en">
<head><meta name="generator" content="Hexo 3.8.0">
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="google-site-verification" content="xBT4GhYoi5qRD5tr338pgPM5OWHHIDR6mNg1a3euekI">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="description" content="威龙的博客">
    <meta name="keyword" content="随缘">
    <link rel="shortcut icon" href="/img/favicon.ico">

    <title>
        
        MooseFs集群故障恢复运维 - 威龙的博客 | VeyronC&#39;s Blog
        
    </title>

    <!-- Custom CSS -->
    <link rel="stylesheet" href="/css/aircloud.css">
    <link rel="stylesheet" href="/css/gitment.css">
    <!--<link rel="stylesheet" href="https://imsun.github.io/gitment/style/default.css">-->
    <link href="//at.alicdn.com/t/font_620856_pl6z7sid89qkt9.css" rel="stylesheet" type="text/css">
    <!-- ga & ba script hoook -->
    <script></script>
</head>

<body>

<div class="site-nav-toggle" id="site-nav-toggle">
    <button>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
    </button>
</div>

<div class="index-about">
    <i> 心若水， 自非凡。该博客已迁移至 https://www.cnblogs.com/xrszff </i>
</div>

<div class="index-container">
    
    <div class="index-left">
        
<div class="nav" id="nav">
    <div class="avatar-name">
        <div class="avatar">
            <img src="/img/avatar.jpg">
        </div>
        <div class="name">
            <i>威龙</i>
        </div>
    </div>
    <div class="contents" id="nav-content">
        <ul>
            <li>
                <a href="/">
                    <i class="iconfont icon-shouye1"></i>
                    <span>主页</span>
                </a>
            </li>
            <li>
                <a href="/tags">
                    <i class="iconfont icon-biaoqian1"></i>
                    <span>标签</span>
                </a>
            </li>
            <li>
                <a href="/archive">
                    <i class="iconfont icon-guidang2"></i>
                    <span>存档</span>
                </a>
            </li>
            <li>
                <a href="/about/">
                    <i class="iconfont icon-guanyu2"></i>
                    <span>关于</span>
                </a>
            </li>
            
            <li>
                <a id="search">
                    <i class="iconfont icon-sousuo1"></i>
                    <span>搜索</span>
                </a>
            </li>
            
        </ul>
    </div>
    
        <div id="toc" class="toc-article">
    <ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#master挂了怎么办"><span class="toc-text">master挂了怎么办</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#当混布chunkserver和metalogger的机器挂了"><span class="toc-text">当混布chunkserver和metalogger的机器挂了</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#测试完集群mfs服务，重新刷一遍-（稳定运行后别再刷）"><span class="toc-text">测试完集群mfs服务，重新刷一遍 （稳定运行后别再刷）</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#若报错找不到-metadata-mfs"><span class="toc-text">若报错找不到 metadata.mfs</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#若启动chunkserver报错不能使用-metaid"><span class="toc-text">若启动chunkserver报错不能使用.metaid</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#读写速率测试"><span class="toc-text">读写速率测试</span></a></li></ol>
</div>
    
</div>


<div class="search-field" id="search-field">
    <div class="search-container">
        <div class="search-input">
            <span id="esc-search"> <i class="icon-fanhui iconfont"></i></span>
            <input id="search-input">
            <span id="begin-search">搜索</span>
        </div>
        <div style="padding:10px 20px;font=10;">
            若搜索无反应请更换浏览器~
        </div>
        <div class="search-result-container" id="search-result-container">

        </div>
    </div>
</div>
        <div class="index-about-mobile">
            <i> 心若水， 自非凡。该博客已迁移至 https://www.cnblogs.com/xrszff </i>
        </div>
    </div>
    
    <div class="index-middle">
        <!-- Main Content -->
        


<div class="post-container">
    <div class="post-title">
        MooseFs集群故障恢复运维
    </div>

    <div class="post-meta">
        <span class="attr">发布于：<span>2019-03-29 14:58:50</span></span>
        
        <span class="attr">标签：/
        
        <a class="tag" href="/tags/#分布式文件系统" title="分布式文件系统">分布式文件系统</a>
        <span>/</span>
        
        
        </span>
        <span class="attr">访问：<span id="busuanzi_value_page_pv"></span>
</span>

    </div>
    <div class="post-content no-indent">
        <p>批量操作集群可以借助<code>pssh</code>，封装之<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">prun: aliased <span class="keyword">to</span> pssh -O <span class="attribute">StrictHostKeyChecking</span>=<span class="literal">no</span> -t 0 -p 90 -h hosts -l work -o out -e err</span><br></pre></td></tr></table></figure></p>
<h1 id="master挂了怎么办"><a href="#master挂了怎么办" class="headerlink" title="master挂了怎么办"></a>master挂了怎么办</h1><p>如果机器能启动起来，则恢复之。<br>如果机器完全崩了，想快速换另一台master，则</p>
<p>用其中一台metalogger服务器来恢复，因为是全混布，都有metalogger，所以选哪台都行。</p>
<p>假设master机器<code>10.46.17.17</code>挂了，要换<code>10.46.19.20</code>来当master<br>在gzns机器的<code>/home/work/pssh/nfs/mfs_conf</code>中修改<code>mfschunkserver.cfg</code>和<code>mfsmetalogger.cfg</code>连接master配置<br><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">MASTER_HOST = <span class="number">10.46</span><span class="number">.19</span><span class="number">.20</span></span><br></pre></td></tr></table></figure></p>
<p>进去gzns机器的zsh状态，批量修改集群的上述两个配置。<br><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="attribute">prun</span> <span class="string">"cd /home/work/.jumbo/etc/mfs &amp;&amp; rm mfschunkserver.cfg; wget ftp://**/home/work/pssh/nfs/mfs_conf/mfschunkserver.cfg"</span></span><br><span class="line">prun <span class="string">"cd /home/work/.jumbo/etc/mfs &amp;&amp; rm mfsmetalogger.cfg; wget ftp://**/home/work/pssh/nfs/mfs_conf/mfsmetalogger.cfg"</span></span><br></pre></td></tr></table></figure></p>
<p>检查 10.46.19.20 机器的<code>mfsmaster.cfg</code>和<code>mfsexports.cfg</code>配置是否存在且配置正确。如果不正确重新刷，或者单台修改。<br><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="attribute">prun</span> <span class="string">"cd /home/work/.jumbo/etc/mfs &amp;&amp; rm mfsmaster.cfg; wget ftp://**/home/work/pssh/nfs/mfs_conf/mfsmaster.cfg"</span></span><br><span class="line">prun <span class="string">"cd /home/work/.jumbo/etc/mfs &amp;&amp; rm mfsexports.cfg; wget ftp://**/home/work/pssh/nfs/mfs_conf/mfsexports.cfg"</span></span><br></pre></td></tr></table></figure></p>
<p>利用<code>mfsmaster -a</code>来合并元数据日志，自动恢复模式。</p>
<p>重启chunkserver和metalogger，如果不重启，可能客户端没法读原来文件，只能写新的，有问题。<br><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="attribute">prun</span> <span class="string">"ps aux | grep mfschunkserver | grep /home/work/.jumbo/sbin/ | awk '&#123;print <span class="variable">$2</span>&#125;' | xargs kill -9"</span></span><br><span class="line">prun <span class="string">"ps aux | grep mfsmetalogger | grep /home/work/.jumbo/sbin/ | awk '&#123;print <span class="variable">$2</span>&#125;' | xargs kill -9"</span></span><br><span class="line">prun <span class="string">"/home/work/.jumbo/sbin/mfsmetalogger start"</span></span><br><span class="line">prun <span class="string">"/home/work/.jumbo/sbin/mfschunkserver start"</span></span><br></pre></td></tr></table></figure></p>
<p>上面kill命令批量报<code>[FAILURE]</code>没事，其实已经执行成功了。</p>
<p>用<code>fusermount -u</code>卸载 因为是fuse挂载的。如果没有卸载成功，在客户端检查下是否还有挂载mfs进程，如果有杀之。<br><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">fusermount -u <span class="regexp">/mnt/m</span>fs</span><br></pre></td></tr></table></figure></p>
<p>挂载新的master的mfs服务<br><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mfsmount -H <span class="number">10.46</span><span class="number">.19</span><span class="number">.20</span> /mnt/mfs</span><br></pre></td></tr></table></figure></p>
<h1 id="当混布chunkserver和metalogger的机器挂了"><a href="#当混布chunkserver和metalogger的机器挂了" class="headerlink" title="当混布chunkserver和metalogger的机器挂了"></a>当混布chunkserver和metalogger的机器挂了</h1><p><a href="https://www.cnblogs.com/bugutian/p/6869278.html" target="_blank" rel="noopener">https://www.cnblogs.com/bugutian/p/6869278.html</a><br>没有备份副本肯定就是endangered的了，如果你的所有磁盘都坏了，那必然就是missing等等。moosefs自己有副本平衡的策略，会自动让你的副本数向对角线靠拢。</p>
<p>chunkserver服务器中，存储的<code>.mfs</code>文件没了的话，大概2分钟会自动备份到别的chunkserver服务器上。</p>
<p>当一台chunkserver不发连接后，mfs会在别的服务器上备份副本到指定数目，等那台故障chunkerserver恢复后，多出来的副本也会自动删除直到最后打平回设置副本数。</p>
<h1 id="测试完集群mfs服务，重新刷一遍-（稳定运行后别再刷）"><a href="#测试完集群mfs服务，重新刷一遍-（稳定运行后别再刷）" class="headerlink" title="测试完集群mfs服务，重新刷一遍 （稳定运行后别再刷）"></a>测试完集群mfs服务，重新刷一遍 （稳定运行后别再刷）</h1><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="attribute">prun</span> <span class="string">"rm -rf /home/work/.jumbo/var/mfs/*"</span></span><br><span class="line">prun <span class="string">"rm -rf /home/disk1/mfs/*"</span></span><br><span class="line">prun <span class="string">"rm -rf /home/disk2/mfs/*"</span></span><br><span class="line">prun <span class="string">"rm -rf /home/disk3/mfs/*"</span></span><br></pre></td></tr></table></figure>
<h2 id="若报错找不到-metadata-mfs"><a href="#若报错找不到-metadata-mfs" class="headerlink" title="若报错找不到 metadata.mfs"></a>若报错找不到 metadata.mfs</h2><figure class="highlight erlang-repl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">loading metadata ...</span><br><span class="line">can't find metadata.mfs - try using option <span class="string">'-a'</span></span><br><span class="line">init: metadata manager failed !!!</span><br></pre></td></tr></table></figure>
<p>在<code>/home/work/.jumbo/var/mfs</code>中拷贝<code>metadata.mfs.empty</code>（若被删了，可重新安装moosefs，或者自己建） 为 <code>metadata.mfs</code>，其内容是：<br><figure class="highlight ada"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">MFSM <span class="keyword">NEW</span></span><br></pre></td></tr></table></figure></p>
<p>再正常启动mfsmaster<br><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="regexp">/home/</span>work<span class="regexp">/.jumbo/</span>sbin<span class="regexp">/mfsmaster start</span></span><br></pre></td></tr></table></figure></p>
<h2 id="若启动chunkserver报错不能使用-metaid"><a href="#若启动chunkserver报错不能使用-metaid" class="headerlink" title="若启动chunkserver报错不能使用.metaid"></a>若启动chunkserver报错不能使用.metaid</h2><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hdd space manager: chunkserver without meta id shouldn't <span class="keyword">use</span> drive <span class="keyword">with</span> defined meta <span class="keyword">id</span> (<span class="keyword">file</span>: <span class="string">'/home/disk1/mfs/.metaid'</span>) - <span class="keyword">use</span> <span class="string">'!'</span> <span class="keyword">in</span> drive definition <span class="keyword">to</span> <span class="keyword">ignore</span> this (dangerous)</span><br></pre></td></tr></table></figure>
<p>先删了.metaid，再启动chunkserver。<br><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="attribute">prun</span> <span class="string">"rm /home/disk1/mfs/.metaid"</span></span><br><span class="line">prun <span class="string">"rm /home/disk2/mfs/.metaid"</span></span><br><span class="line">prun <span class="string">"rm /home/disk3/mfs/.metaid"</span></span><br></pre></td></tr></table></figure></p>
<h1 id="读写速率测试"><a href="#读写速率测试" class="headerlink" title="读写速率测试"></a>读写速率测试</h1><p>副本数和metalogger增加基本不影响写速率。</p>
<p>若指定goal高于2，客户端将数据发送到一个组块服务器，组块服务器读取，写入数据并将数据发送到另一个组块服务器。这样一来，客户端的输入就不会发送多个副本，而且所有副本几乎同时被写入。</p>
<p>用1.5G的包含大量小文件整个目录写入测试，2m14s，速率大约11M/s。删除耗时42s，速率大约37M/s。</p>
<p>429M大文件写入耗时3.89s，速率大约110M/s。</p>

        
            <div class="donate-container">
    <div class="donate-button">
        <button id="donate-button">赞赏</button>
    </div>
    <div class="donate-img-container hide" id="donate-img-container">
        <img id="donate-img" src data-src="/img/donate.jpg">
        <p> 感谢鼓励 </p>
    </div>
</div>
        
        <br>
        <div id="comment-container">
        </div>
        <div id="disqus_thread"></div>

        <div id="lv-container">
        </div>

    </div>
</div>

    </div>
</div>


<footer class="footer">
    <ul class="list-inline text-center">
        
        

        
        <li>
            <a target="_blank" href="http://weibo.com/2983694150">
                            <span class="fa-stack fa-lg">
                                  <i class="iconfont icon-weibo"></i>
                            </span>
            </a>
        </li>
        

        

        

        

    </ul>
    
    <p>
        <span>/</span>
        
        <span><a href="https://www.10000h.top">10000H</a></span>
        <span>/</span>
        
        <span><a href="https://niexiaotao.com">Xiaotao&#39;s Page</a></span>
        <span>/</span>
        
        <span><a href="#">It helps SEO</a></span>
        <span>/</span>
        
    </p>
    
    <p>
        <span id="busuanzi_container_site_pv">
            <span id="busuanzi_value_site_pv"></span>PV
        </span>
        <span id="busuanzi_container_site_uv">
            <span id="busuanzi_value_site_uv"></span>UV
        </span>
        Created By <a href="https://hexo.io/">Hexo</a>  Theme <a href="https://github.com/aircloud/hexo-theme-aircloud">AirCloud</a></p>
</footer>




</body>

<script>
    // We expose some of the variables needed by the front end
    window.hexo_search_path = "search.json"
    window.hexo_root = "/"
    window.isPost = true
</script>
<script src="https://cdn.bootcss.com/jquery/3.3.1/jquery.min.js"></script>
<script src="/js/index.js"></script>
<script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script>


    <script>
        /**
         *  RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
         *  LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables
        */
        if( '' || '')
        var disqus_config = function () {
            this.page.url = '';  // Replace PAGE_URL with your page's canonical URL variable
            this.page.identifier = ''; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
        };

        (function() { // DON'T EDIT BELOW THIS LINE
            var d = document, s = d.createElement('script');
            s.src = 'https://airclouds-blog.disqus.com/embed.js';
            s.setAttribute('data-timestamp', +new Date());
            (d.head || d.body).appendChild(s);
        })();
    </script>



</html>
